Skip to content

[models] Upgrade Qwen3.5 implementation to Qwen3.8, supports text-only path - #4355

Merged
wwwjn merged 9 commits into
pytorch:mainfrom
wwwjn:upgrade-qwen3-8
Sep 2, 2026
Merged

[models] Upgrade Qwen3.5 implementation to Qwen3.8, supports text-only path#4355
wwwjn merged 9 commits into
pytorch:mainfrom
wwwjn:upgrade-qwen3-8

Conversation

@wwwjn

@wwwjn wwwjn commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace the qwen3_5 model package with a single qwen3_8 package
  • support the released Qwen3.8-27B multimodal model and Qwen3.8-2.4T-A95B text-only MoE model
  • remove all legacy Qwen3.5 model flavors and recipe configurations
  • support both Hugging Face checkpoint layouts: model.language_model.* for the multimodal checkpoint and model.* for the text-only checkpoint
  • make the vision encoder optional for the text-only MoE model
  • rename Qwen tests, numerical utilities, integration recipes, and RL entry points
  • explicitly exclude Qwen3.8-Flash-Next because it uses the separate qwen4_exp architecture

Hugging Face retains the Qwen3_5* implementation class and model-type names for these Qwen3.8 checkpoints, so those internal class names remain where they describe the upstream architecture contract.

Qwen3.8-27B numerical parity

Tested against Qwen/Qwen3.8-27B with Transformers 5.15 on an NVIDIA H100. The released 55.6 GB safetensor checkpoint was converted through the TorchTitan state-dict adapter to DCP without missing-key or shape errors.

FP16 multimodal, three deterministic image-text samples

  • average KL divergence: 1.6405e-6
  • cosine similarity: approximately 0.99997
  • top-1 match: 100%
  • top-5 match: 100%
  • image preprocessing: all 589824 reconstructed pixel values matched, maximum difference 1.19e-7

Per-sample KL: 2.1510e-6, 4.8427e-7, 2.2861e-6.

Text-only FP16 isolation

  • KL divergence: 4.3627e-7
  • cosine similarity: 0.999966
  • top-1/top-5 match: 100%

This isolates and validates the decoder architecture and HF weight conversion independently of the vision encoder.

BF16 multimodal, three deterministic samples

  • average KL divergence: 1.9280e-4
  • top-1/top-5 match: 100%

Tests

  • uvx pre-commit run --all-files
    • all formatting, lint, docstring, spelling, and link hooks pass
    • repository-wide Pyrefly reports three pre-existing environment/API errors outside this change
  • scoped Pyrefly check for the changed Qwen3.8 implementation and parity utilities: pass
  • pytest -q tests/unit_tests/cpu/test_qwen3_8.py tests/unit_tests/cpu/test_no_new_cli_options.py::TestCliOptionsFrozen::test_every_model_is_guarded tests/unit_tests/cpu/components/data/test_qwen_multimodal_data.py::test_qwen38_recipe_geometry_matches_dataset_processor
    • 13 passed

Some existing varlen-attention tests require the separately installed FlashAttention-3 flash_attn_interface module.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 27, 2026
@wwwjn
wwwjn requested a review from shuhuayu August 27, 2026 20:21
@shuhuayu

shuhuayu commented Aug 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

do we want to retire qwen 3.5 now or we can in addition support qwen 3.8 in the same folder? what are the major differences?

tianyu-l
tianyu-l reviewed Aug 27, 2026
View reviewed changes
Comment thread torchtitan/models/qwen3_5/__init__.py
@wwwjn

wwwjn commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

do we want to retire qwen 3.5 now or we can in addition support qwen 3.8 in the same folder? what are the major differences?

My current plan is to retire all qwen3.5 and just keep Qwen 3.8 model. However qwen3.8 only have 27B model as the smallest model.

Because the model architecture is the same, we can keep these small models in the config_regitry.py as well?

@tianyu-l

tianyu-l commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Because the model architecture is the same, we can keep these small models in the config_regitry.py as well?

I think we should keep two folders one for 3.5 and one for 3.8.

  • Most model code can be in one of them (e.g. we can put in 3.8 so that later we can remove 3.5 first)
  • __init__.py file has the model_registry we need to keep in both.

@wwwjn wwwjn changed the title [models] Upgrade Qwen3.5 implementation to Qwen3.8 [WIP[models] Upgrade Qwen3.5 implementation to Qwen3.8 Sep 1, 2026
@wwwjn wwwjn changed the title [WIP[models] Upgrade Qwen3.5 implementation to Qwen3.8 [WIP][models] Upgrade Qwen3.5 implementation to Qwen3.8 Sep 1, 2026
@wwwjn

wwwjn commented Sep 1, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Updated per the follow-up: qwen3_5 is again the canonical shared implementation package (model, GatedDeltaNet, RoPE, vision encoder, sharding, parallelization, state-dict adapter, and existing tests/scripts). qwen3_8 is now lightweight and contains only its version-specific registry, recipes, and README while reusing the Qwen35 implementation names. The Qwen3.5 small dense and MoE flavors remain in their original registry.

wwwjn
wwwjn commented Sep 1, 2026
View reviewed changes
Comment thread torchtitan/models/qwen3_5/model.py

self.vision_encoder = config.vision_encoder.build()
self.spatial_merge_size = config.vision_encoder.spatial_merge_size
self.vision_encoder = (

@wwwjn wwwjn Sep 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Qwen3.8-2.4T-A95B model is text only so we want to allow None vision_encoder

@wwwjn wwwjn changed the title [WIP][models] Upgrade Qwen3.5 implementation to Qwen3.8 [models] Upgrade Qwen3.5 implementation to Qwen3.8, supports text-only path Sep 1, 2026
tianyu-l
tianyu-l approved these changes Sep 2, 2026
View reviewed changes

@tianyu-l tianyu-l left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. need to be careful when you change model size in config_registry
  2. should we rename 3.5 to 3.6 if it also has all the small variants (don't if not)

Comment thread torchtitan/experiments/rl/examples/alphabet_sort/config_registry.py
)


def rl_grpo_qwen3_5_9b_varlen() -> Controller.Config:

@tianyu-l tianyu-l Sep 2, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't change this? model size change, then parallelism / ac settings may be invalidated

@wwwjn wwwjn Sep 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, let me create a separate 27B RL config and revert these

Qwen3.6 has the same structure as qwen3.5, but qwen3.5 has more size: 0.8B, 2B, 4B, 9B, 27B, 35B, 122B, 300B. Qwen3.6 only has 27B and 35B size, not smaller size model. Should we add another folder for Qwen3.6?

Update: I will just don't rename qwen3_5 to 3_6 as this moment, and don't add 3.6 model config (seems add little value there)

Comment thread torchtitan/experiments/rl/tests/test_bitwise_parity.py Outdated
@wwwjn
wwwjn merged commit 0ddd970 into pytorch:main Sep 2, 2026
12 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tianyu-l tianyu-l tianyu-l approved these changes
@fegin fegin Awaiting requested review from fegin fegin is a code owner
@wconstab wconstab Awaiting requested review from wconstab wconstab is a code owner
@shuhuayu shuhuayu Awaiting requested review from shuhuayu

Assignees

No one assigned

Labels

ciflow/rl CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@wwwjn @shuhuayu @tianyu-l